Hive: Fix predicate pushdown for Timestamp.withZone() #2278

pvary · 2021-02-26T15:06:18Z

During #2254 we postponed to create tests for tables where the column type is timestamp.withZone().
This PR adds them with the appropriate tests.

The premise:

Hive2 uses sql.Timestamp internally - which is in the LocalTimezone
Hive3 uses hive.Timestamp internally - which is in UTC
Iceberg timestamp.withZone() stores the values in OffsetDateTime in UTC

What I have done:

Fixed the conversions for IcebergTimestampWithZoneObjectInspector
Simplified a code a little bit for IcebergTimestampObjectInspectorHive3
Added the junit tests for SELECT
- Had to add one more magic setting so the TimeZone is handled correctly
Added additional check to the existing unit tests to make sure that we get back the same string in SQL as we insert
Had to change the unit tests for Hive2, so the expected value in the Iceberg record is considers the TimeZone

(cherry picked from commit 18060d71f5fa3f58ed79c11dd4399c209b115c3c)

mr/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergTestUtils.java

mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java

...rg/apache/iceberg/mr/hive/serde/objectinspector/IcebergTimestampWithZoneObjectInspector.java

pvary · 2021-03-01T16:21:45Z

@shardulm94: Could you please review this PR? This is the TimestampTz part of our previous change (#2254)

Thanks,
Peter

github-actions · 2024-07-27T00:13:12Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions · 2024-08-03T00:13:22Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

Hive: Fix predicate pushdown for Timestamp.withZone()

1e4412e

(cherry picked from commit 18060d71f5fa3f58ed79c11dd4399c209b115c3c)

github-actions bot added hive MR labels Feb 26, 2021

This was referenced Feb 26, 2021

Hive: Fix predicate pushdown for Date #2254

Merged

Revert "Hive: Fix predicate pushdown for Date (#2254)" #2279

Closed